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Abstract 

The research identifies the complexity level of eight texts from Spotlight 11 used in Russian 
TEFL to prepare students for National Unified Exam in English and assess their reading skills. The results 
of the analyses conducted with the help of T.E.R.A., an automated text processor, prove that all texts fell 
within the range of 6 - 9 Flesch-Kincaid grade levels which correspond to the English language 
proficiency of the target audience. We also revealed the absence of a clear progression in difficulty across 
the eight texts in the continuum which may cause unpredictable test results and contribute to de- 
motivation of students. The results also show that the indices of narrativity, syntactic simplicity, word 
concreteness, referential cohesion and deep cohesion measured with T.E.R.A. do not grow but fluctuate 
across the continuum of the texts either. Aiming at selecting authentic texts with steadily growing 
complexity of each of the above mentioned parameters, we recommend to incorporate the suggested 
algorithm of text analysis into TEFL practice in Russia. T.E.R.A. is viewed by the authors as a tool able 
to provide educators with a solid foundation to select texts, develop curriculum, design assessment tasks 
and otherwise address academic needs of a target audience. 
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Introduction 

The problem of fostering reading comprehension is never going to cease its significance 
to education as it is a key feature to any academic success. The journey from primary school to 
college and along one’s career path is a reading journey with reading texts becoming more 
complex in terms of both concepts and language (Erbilgin, 2017; Pearson & Liben, 2015; Mauch 
& Tarman, 2016; Tarman, 2016). If textbooks offered to schoolchildren for reading 
comprehension do not correspond to children’s cognitive and linguistic abilities, it often results 
in reading tasks losing their attractiveness for children, frustrations and lack of interest in studies 
(Gabitov & Ilyasova, 2016). Describing the current situation in TEFL in the Russian Federation, 
Solnyshkina and Kiselnikov (2015) argue that while selecting texts for educational purposes 
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Russian textbooks writers and exam developers measure predominantly ‘text readability’ and use 
numerical scores obtained with the existing readability formulas, i.e. Flesch Reading Ease, 
Flesch-Kincaid Grade Level, Gunning Fog Score etc. The consequences of this oversimplified 
approach to the problem are inadequate texts used in English language teaching and testing in the 
Russian Federation. 

One of the modern automated tools designed for texts complexity parameters analysis is 
T.E.R.A., Coh-Metrix Common Core Text Ease and Readability Assessor measuring scores of 
narrativity, syntactic simplicity, word concreteness, referential cohesion, deep cohesion and 
Flesch-Kincaid Grade level. Thus, providing a user with a comprehensive picture of text 
characteristics, T.E.R.A. simplifies the process of selecting texts for potential readers. 

Methods 

The existing readability formulas correlating text readability with school grades are based 
on the two main variables, i.e. mean number of letters/syllables per word and words per sentence 
(Readability Formulas. Free readability tools to check for Reading Levels, Reading Assessment, 
and Reading Grade Levels). For example, Flesch-Kincaid Grade Level (F-K GL) is calculated in 
a standard way using sentence length (in words) and word length (in letters) as follows: (.39 * 
sentence length) + (11,8 * word length) - 15,59. An increase in one of the two components of the 
formula results in a higher value of the text readability. Though the readability scores are widely 
used in pedagogical community all over the world (Duran et al., 2007), their accuracy as well as 
ability to correlate with children’s cognitive age have been a persistent concern of many 
educators since the formulas were first produced (Pearson & Liben, 2015). The researchers argue 
that the formulas do not estimate a number of text features, such as lexical variety, complexity of 
grammatical structures and/or general logic of textual information. All these make them an 
extremely limited means of assessing text appropriateness for a target audience (Duran et al., 
2007). 

Another stage of the studies in the area was marked with the extension of the range of 
text parameters that assess their complexity (Solnyshkina & Kiselnikov, 2015). In the review 
article by Kiselnikov (2013) the authors conclude that the majority of modern text complexity 
criteria are primarily derived either based on syntactic or lexical features of texts. As a semantic 
category, text complexity realizes through grammar and vocabulary in a text. 



Solnyshkina et al. 


T.E.R.A marked the transition to another stage of the problem development as it also 
measures text cohesion and coherence. At the moment, TERA, available on the public website, 
calculates Flesch-Kincaid Grade Level (F-K GL) and computes five characteristics of texts: 
syntactic simplicity, abstractness/concreteness of words, narrativity, referential cohesion, deep 
cohesion. 

For the study presented we computed the text parameters, such as Flesch-Kincaid Grade 
levels, narrativity, syntactic simplicity, abstractness/concreteness of words, referential cohesion, 
deep cohesion of the texts from Spotlight 11, with the help of T.E.R.A and predominantly used 
descriptive, interpretative and contrastive methods to identify to what degree the texts 
correspond the cognitive and linguistic levels of the Russian students. 

Results 

We selected the data for the study from the students book Spotlight 11 recommended by 
the Ministry of Education of the Russian Federations for English language teaching in the 11 th 
grade of public schools. The 11 th grade is the final year in Russian high schools during which 
students are predominantly trained for final matriculation examination. To ensure the 
equivalence of text materials in terms of the purpose of learning, all the texts for the study were 
selected from Chapters A of each module of the textbook. Chapters A are designed for testing 
the students’ skimming and scanning in the classroom. The texts were attributed numbers 1-8 
and letter A indicating the corresponding Module and Chapter. E.g. Text 2A stands for the text 
used in Spotlightl 1 to assess students’ reading skills in Module 2, Chapter A. 

The genres of the texts include the following: realistic fiction, science fiction, biographies 
and magazine articles. All the texts selected are narrative expect for Text 8A which is 
descriptive. The word count varies from 385 in text 3A to 657 in text 8A. The mean number of 
sentences is 29. The texts were formatted in .txt files and assessed in a single-text format with 
the help of T.E.R.A. for narrativity, syntactic simplicity, abstractness/concreteness of words, 
referential cohesion, deep cohesion. 

On the first stage of the analysis we used Flesch-Kincaid formula to calculate the 
corresponding reading grade level of texts 1A - 8A, i.e. what reading age children they are 
appropriate for. The results, as demonstrated on the graph in Img.l below, testify to the absence 
of a linear progression of F - K Grade Level with texts 3A, 5A suitable for Grade 6 and texts 8A, 
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6A being appropriate for Grade 9 of USA schools. The easiest is Text 5A with F-K GL 6,20 and 
the most difficult text is Text 6A with F-K GL 9,70 (see Img. 1 below). 



Image 1. Flesh-Kincaid Grade Levels Graph for texts 1A - 8 A. 


On the second stage of the research we computed the 8 selected texts with the help of 
T.E.R.A. aiming at measuring text complexity parameters and each of its characteristics, i.e. of 
narrativity, syntactic simplicity, abstractness/concreteness of words, referential cohesion and 
deep cohesion. 

Avoiding restrictive categorizing of narrativity, T.E.R.A developers adapt a comparative 
view on the notion and define narrativity in relative terms: a text may have higher or lower 
degree of narrativity in contrast with all the texts in T.E.R.A. library. In general terms narrativity 
is defined as the quality or condition of presenting a narrative (Oxford Living Dictionaries). 
Accordingly, texts with a low value of narrative parameters contain a high proportion of unique 
nouns and a variety of temporal forms as is the case in Text 8A: “ Otherwise known as The Lost 
City of the Incas', Machu Picchu is an ancient Incan city located almost 2,500 metres above sea 
level in the Andes Mountains in Peru. Machu Picchu is invisible from below An ideally simple 
text will be a set of similar type sentences, whereas a complex one will not have one similar 
combination of pairs of sentences in its entire body. “The identity of sentences in the text will, 
however, change the deep connectivity of the text” (Crossley & McNamara, 2016). 
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All the texts of Chapter A, except for Text 8A, are narrative and the corresponding 
parameter scores measured by T.E.R.A. vary from minimum of 69% in Text 4 to maximum of 
92% in Text 3A. Text 3A demonstrates a high density of verbs: “I couldn't even remember 
buying a ticket! Anyway, I contacted the people the letter was from, who asked me to send them a 
fee' to process my winnings 

The Narrativity of Text 8A, as shown in Img. 2, is only 30%. Thus, T.E.R.A. 
discriminated the text of another genre by calculating its Narrativity. 


r a 



v_ j 

Image 2. Narrativity Graph for texts 1A - 8 A. 


Though indices of narrativity of Texts 1A - 7A do not progress form low to high, they 
demonstrate a certain degree of consistency and do not fall below 69%. Text 8A was excluded 
from the further analysis based on the assumption that different types of texts demonstrate 
differences in the frequency of core vocabulary words (Lee, 2001), in the way that cohesion 
relations are expressed (McCarthy, Graesser & McNamara, 2006), in the rate at which rare 
words are repeated (Heibert, 2009), etc. 

Indices of syntactic simplicity, word concreteness, referential cohesion, deep cohesion for 
texts 1A - 7A are presented on the corresponding graphs in Img.3 below. The observed 
fluctuations are as follows: for Syntactic Simplicity - 31%, Word Concreteness - 64%, 
Referential Cohesion - 31%, with the minimum variation in Deep Cohesion - 25%. The 
Syntactic Simplicity graph demonstrates that the syntax indices of the texts studied are medium 
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and vary from 34% in Text 1A to 65 % in texts 2A and 4A. The lowest indices are those of 
Referential Cohesion , with the lowest of 9 % in Text 7A and the highest being 40% in text 3A. 
Texts also showed a decrease in Referential Cohesion from around 40 % in Texts 1 A and 2A to 
9% in Text 7A. But there is a remarkable growth in the numbers observed in Word 
Concreteness : 14 % in Text 6A and 70% in Text 3A. Cf.: “The searches are based on the 
hypothesis that aliens may be intentionally sending communication signals out across the 
universe, or that signals from their worlds could be escaping into space - just as our own radio, 
television and military broadcast signals escape into the cosmos” (14 %, Text 6A); “Burglars 
recently broke into our house while we were sleeping upstairs! My sister and I heard a noise, so 
we woke up our dad, who called the police.” (70%, Text 3A). Deep Cohesion of all the texts 
studied is relatively high and remains between 74% in Text 3A and 99% in Text 2A. 


Indices of Text Parameters Measured with T.E.R.A. 



Image 3. Graphs of Complexity Parameters for Texts 1A - 7A. 

Discussion 

Syntactic simplicity is a parameter dependent on three measured variables, i.e. mean of 
clauses throughout the text, mean of words in the sentence, and mean of words in front of the 
main verb (McNamara & Graesser, 2012). Table 1 below shows the complexity parameters of 
the texts with minimum (1A) and maximum (2A) scores of Syntactic simplicity. As we see their 
F-K G Levels are only 1.3 grade different. The excepts from Text 1A and Text 2A below are 
self-explanatory. Cf. Text 1 :1 am an only child and 1 live with my parents and my grandma, or 




Solnyshkina et al. 


'babushka' as we say here in Russia. My grandpa passed away last year so Babushka left her 
house in the country to come and live with us here in the city. Text 2A: In a nutshell, "stress" is 
the way our bodies and minds react to life's changes. In stressful situations, the nervous system 
causes muscles to tense, breathing to become shallow and adrenaline to be released into your 
bloodstream as your body gets ready to beat challenges with focus and strength. 


Table 1 


Indices for Two Representative Texts 1A and 7A 


Text 

Narrativity 

Syntactic 

Word 

Referential 

Deep 

Flesh -Kincaid 



simplicity 

Concreteness 

Cohesion 

Cohesion 

Grade Level 

1 

79% 

34% 

36% 

39% 

81% 

8,20 

2 

77% 

65% 

39% 

37% 

99% 

7,40 


Abstractness/ Concreteness of words as it comes from the name, shows the proportion of 
concrete words to abstract ones. The developers of T.E.R.A. define concrete words as “words 
that refer to things you can see, hear, taste, touch, feel, or smell. Abstract words cannot easily be 
seen, heard, touched, felt or smelled” (McNamara & Graesser, 2012). With a high content of 
specific vocabulary or professional jargon, the text becomes more difficult for the reader to 
understand. The texts studied contain a number of words the concreteness of which is obvious, 
i.e. sand, beaches, friends, house, dad (Texts 3A, 6A). The abstract words registered in the 
corpus of the texts Spotlight 11 are as follows: dream, obstacles, failure, gocd, ambitions (Text 
7A). But category of Abstractness/ Concreteness is a non clear-cut phenomenon but a 
continuum, and there are words in the vocabulary of every language which depending on the 
context may be viewed either as concrete or abstract: ‘a field of tulips’ vs ‘a field of knowledge’. 
The potential difficulty in measuring the parameter of Abstractness/ Concreteness in a separate 
text lies in the requirement to identify its value for every word in the text, which implies the 
necessity of a source referring to which a researcher would get the value of each word in a text. 
T.E.R.A. developers use MRC Psycholinguistic database for concreteness ratings, the word lists 
of which “are incomplete due to the limited size of the word samples” (McNamara & Graesser, 
2012). The texts with extreme indices of Word Concretenessin the continuum are Texts 5A and 
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6A with F-K GL 6.20 and 9.70, respectively (See Table 2). The majority of the indices of the 
texts are distinctively different while F-K G Levels are about the same. 


Table 2 


Indices for Two Representative Texts 5A and 6A 


Text 

Narrativity 

Syntactic 

simplicity 

Word 

Concreteness 

Referential 

Cohesion 

Deep 

Cohesion 

Flesh -Kincaid 

Grade Level 

5 

80% 

55% 

78% 

13% 

94% 

6,20 

6 

75% 

51% 

14% 

20% 

94% 

9,70 


Referential cohesion is a measure of the overlap between words in the text, formed with 
the help of similar words and ideas transmitted by them (McCarthy et al., 2006). To maintain a 
high level of the parameter, it is necessary to provide proposals with one or more types of 
specified means (Rowe, Ozuru & McNamara, 2006). Contrasting indices of Referential cohesion 
in Texts 1A - 7A, we revealed a range of scores with the minimum of 9% in Text 7 A and the 
maximum of 40% in Text 3 A. Consider a sample from Text 3A with a high value of this 
parameter: Getting married and having a family is a top priority for me. My marriage will 
probably be arranged by my father, which is quite common here. (Spotlight 11, Text 1A). We 
can see a number of repetitions of derivatives related to the same root: married - marriage, me - 
my as well as hyperonym - hyponym relations: a family - my father. The word which refers to 
the idea of ‘the marriage being arranged by the father of the writer’ thus connecting two parts of 
the sentenceChat the reader interprets as a whole. Text 7A demonstrating the lowest value of 
Referential cohesion in the corpus, i.e. 9%, runs as follows: 

“Believe in yourself {subtitle of the text). 

Without a doubt, this is the first step on the road to achieving your dreams! And this 
means not listening to anyone who tells you, 'You can't.' Einstein, Beethoven and Edison, cdl 
knew about this It is obvious that the pronoun this in the first sentence of the text refers to the 
idea expressed in the subtitle of the text, i.e. believe in yourself to elicit this information is much 
more cognitively difficult for a non-native student of English than in a sentence with stronger 
connections between parts of a text and a sentence. 
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Table 3 


Indices for Two Representative Texts 6A and 8A 


Text 

Narrativity 

Syntactic 

Word 

Referential 

Deep 

Flesh -Kincaid 



simplicity 

Concreteness 

Cohesion 

Cohesion 

Grade Level 

3 

92% 

54% 

70% 

40% 

74% 

6,50 

7 

84% 

63% 

33% 

9% 

95% 

7,50 


Texts 6A and 8A, i.e. the texts with the highest and lowest values of referential cohesion, 
have a high value of narrativity and deep cohesion but a relatively low value of Referential 
Cohesion. Referential Cohesion scores of 9% certifies to poor logical connections between the 
ideas in the adjacent sentences of the texts, i.e. coherence. 

Deep Cohesion determines how well the events, ideas, and information of the entire text 
are related at a level that is understandable by a human (Readability Formulas. Free readability 
tools to check for Reading Levels, Reading Assessment, and Reading Grade Levels). As it can be 
observed from Table 4, Deep Cohesion has very little correlation with F-K GL. 


Table 4 


Indices Two Representative Texts 2A and 8A 


Text 

Narrativity 

Syntactic 

Word 

Referential 

Deep 

Flesh -Kincaid 



simplicity 

Concreteness 

Cohesion 

Cohesion 

Grade Level 

2 

77% 

65% 

39% 

37% 

99% 

7,40 

8 

30% 

36% 

80% 

22% 

42% 

9,50 


Thus, the research demonstrated a wide range of syntactic simplicity (over 30%) and 
word concreteness scores (60%) in the narrative texts studied. Narrativity and Deep Cohesion 
scores do not fall below 69%, while referential cohesion does not rise above 40%. As we know 
“Automatic event extraction is an important task in knowledge acquisition step” (Solovyev & 
Ivanov, 2016). 

Conclusion 

The comparative study of eight texts used in Spotlight 11 to assess students’ reading 
skills demonstrated that Flesch-Kincaid Grade Levels of the texts fluctuate from 6 to 9 
presenting a non-linear progression. This fact revealing lack of a comprehensive approach 
applied by the authors of Spotlight 1 lto selecting texts for academic purposes may not only de- 
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motivate students but also invalidate the tests in which the texts are used. We advocate 
sequencing academic texts from easy to difficult since it is more likely to generate positive 
responses from students and have a positive impact on reading tests results. 

Findings on Narrativity, Syntactic Simplicity, Word Concreteness, Referential and Deep 
Cohesion variables of Coh-Metrix gained by the authors are similar to those of Flesch-Kincaid 
Grade level - none of the indices demonstrated a progression of easier to harder across the 
continuum of the eight studied texts. As consistent progression of each complexity parameter in 
authentic texts selected for academic purposes, though ideal but inaccessible, we suggest using 
T.E.R.A. as an instrument to help educators to assess text dimensions and provide students with 
enough training to be ready for separate complexity parameters spikes. 
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